mmii 

06/02/00 



Birch, Stewart, Kolasch & Birch, llp 



TERRELL C BIRCH 
RAYMOND C STEWART 
JOSEPH A KOLASCH 
JAMES M SLATTERY 
BERNARD L SWEENEY- 
MICHAEL K MUTTER 
CHARLES GORENSTEIN 
GERALD M MURPHY. JR 



ANDREW D MEIKLE 
MARC S. WEINER 
JOE McKINNEY MUNCY 
ROBERT J KENNEY 
DONALD J DALEY 
JOHN W. BAILEY 
JOHN A. CASTELLANO, III 
GARY D. VACURA 

OF COUNSEL 

HERBERT M BIRCH (1905-1996) 
ELLIOT A GOLDBERG- 
WILLIAM L GATES- 
EDWARD H VALANCE 
RUPERT J BRADY (RET)* 
F. PRINCE BUTLER 
FRED S. WHISENHUNT 



INTELLECTUAL PROPERTY LAW 
8110 Gatehouse Road 
Suite 500 East 
Falls Church, VA 22042-1210 
USA 
(703) 205-8000 



THOMAS S AUCHTERLONIE r>. 
JAMES T ELLER, JR ^ 

SCOTT L LOWE VO^' 

MARK J NUELL, PhD <<^C 

D RICHARD ANDERSON ODC 

PAULC LEWIS U 

MARKW MILSTEAD" ^5 
JOHN CAMPA- 
RICHARD J GALLAGHER 



REG PATENT AGENTS 
FREDERICK R HANDREN 
MARYANNE ARMSTRONG. Ph 0 
MAKI HATSUMI 
MIKE S RYU 
CRAIG A McROBBIE 
GARTH M DAHLEN, Ph D 
LAURA C LUTZ 
ROBERT E GOOZNER. Ph D 
HYUNGN SOHN 
MATTHEW J. LATTIG 
ALAN PEDERSEN-GILES 
JUSTIN D KARJALA 



Date: June 2, 2000 
Docket No . : 3562-0102P 



ftl Assistant Commissioner for Patents 
m Box PATENT APPLICATION 
{H Washington, D.C. 20231 



As authorized by the inventor (s), transmitted herewith for filing 
^ is a patent application applied for on behalf of the inventor (s) 
according to the provisions of 37 CFR 1.41(c). 

JJJ Inventor (s) : ONO, Shuji 

For: IMAGE PROCESSING APPARATUS, IMAGE OUTPUT APPARATUS, AND 

CAMERA 

Enclosed are : 

X A specification consisting of 30 pages 

X 10 sheet (s) of Formal drawings 

X Certified copy of Priority Document (s) 

X Executed Declaration in accordance with 37 CFR 1.64 will follow 

A verified statement to establish small entity status under 37 

CFR 1.9 and 37 CFR 1.27 

Preliminary Amendment 

X Information Sheet 

Information Disclosure Statement, PTO-1449 with reference (s) 



Mail Addre.ss P O Box 747, Falls Church, Vjkginia, USA 22040-0747 



3562-0102P 



Other 

The filing fee has been calculated as shown below: 



LARGE ENTITY SMALL ENTITY 



FOR 


NO. FILED 


NO. EXTRA 


RATE 


FEE 




RATE 




FEE 




BASIC FEE 


^^^^^^^^^^^ 
*********** 
*********** 


^^^^^^^^^^ 
********** 
********** 


***** 
***** 


$690. 


00 


or 


HH 




$345 


00 


TOTAL 
CLAIMS 


21 - 20 = 


1 


xlB =$ 


18. 


00 


or 


X 9 = 


$ 


0 


00 


INDEPENDENT 


8 - 3 = 


5 


x78 =$ 


390. 


00 


or 


X 39 = 


$ 


0 


00 


MULTIPLE DI 
CLAIM PRESI 


]PENDENT 
]NTED nc 


2_ 


+ 260 = 


$ 0. 


00 


or 


+130 = 


$ 


0 


00 






TOTAL $1,098. 


00 




TOTAL 


$ 


0 


00 



X The application transmitted herewith is filed in accordance 

with 37 CFR 1.41(c). The undersigned has been authorized by the 
inventor (s) to file the present application. The original duly 
executed patent application together with the surcharge will be 
forwarded in due course . 

X A check in the amount of S i. 098 .00 to cover the filing fee and 
recording fee (if applicable) is enclosed. 

The Government Filing Fee will be paid at the time of completion 

of the filing requirement. 

Please charge Deposit Account No. 02-2448 in the amount of 

$ . A triplicate copy of this transmittal form is 

enclosed. 



Send Correspondence to: 



BIRCH, STEWART, KOLASCH & BIRCH, LLP 
P. 0. Box 747 

Falls Church, Virginia 22040-0747 



3562-0102P 



No fee is enclosed. 

If necessary, the Commissioner is hereby authorized in this, 
concurrent, and future replies, to charge payment or credit any 
overpayment to Deposit Account No. 02-2448 for any additional 
fees required under 37 CF.R. 1.16 or under 37 C.F.R. 1,17; 
particularly, extension of time fees. 




Reg. No. 29,680 
P. 0. Box 747 

Falls Church, Virginia 22 040-0747 

(703) 205-8000 
MKM/dpt 



1 



IMAGE PROCESSING APPARATUS, IMAGE OUTPUT APPARATUS, AND :;AMERA 

This patent application claims priority based on a Japanese 
patent application, Hll-157158 filed on June 03, 1999, the 
5 contents of which are incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

10 

The present invention relates to an image processing 
apparatus, an image output apparatus and a camera, and more 
particularly to an image processing apparatus, an image output 
apparatus and a camera capable of determining conditions for 
15 processing, outputting and inputting an image based on depth 
information extracted from a parallactic image. 

2. Description of the Related Art 

20 Conventionally, a technique is known to extract image 

characteristics from a photographed image and process the image 
in accordance with the extracted image characteristics. This 
technique is disclosed in Japanese Patent Laid-open Publication 
(Kokai) H7 — 200824. However, information obtained from the image 

25 characteristics is not enough. Therefore, when a lot of people 
are in the image, it is difficult to separately recognize faces 
of people which are partially overlapping. It means that it is 
difficult to process each of the components in the image in 
accordance with different conditions, based on the image 

30 characteristics. 

Japanese Patent Laid-open Publication (Kokai) H5— 196857 
discloses a technique to control exposure of an image based on 
a main object in the image. The main object is extracted from 
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the image by setting a plurality of measuring points, and 
detecting variation of the data at the measuring points per 
predetermined period, using a conventional triangulat ion method. 
This technique uses infrared radiation for measuring the distance 
5 to each of the components in the image 



However, it was difficult to increase the measuring points 
because the directivity of the infrared radiation is limited. 
Therefore, the main object cannot be precisely extracted. 

SUMMARY OF THE INVENTION 



Therefore, it is an object of the present invention to 
provide an image processing apparatus, an image output apparatus 
15 and a camera, which overcome the above issues in the related art. 
This object is achieved by combinations described in the 
independent claims. The dependent claims define further 
advantageous and exemplary combinations of the present 
invention . 

20 

According to the first aspect of the present invention, an 
image processing apparatus for processing an image obtained by 
photographing a subject, comprises a condition-determining unit 
determining a condition for processing the image based on depth 
25 information indicating the distance to each part of the subject. 

The image processing apparatus may include an image 
processing unit processing the image based on the condition for 
processing the image. 

30 

The image processing apparatus may include an extractor 
extracting the depth information based on a parallactic image 
obtained by photographing the subject from different viewpoints. 
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The extractor may extract data of an aimed obj ect from the 
image based on the depth information, and the 
condition- determining unit may determine the condition for 
processing the image based on information including the data of 
5 the aimed object. 

The extractor may extract image information included in the 
image and extract the aimed object based on the image information. 

10 According to the second aspect of the present invention, 

an image output apparatus for output ting an image obtained by 
photographing a subject comprises a condition-determining unit 
determining a condition for outputting the image based on depth 
information indicating the distance to each part of the subject. 

15 

The image output apparatus may include an output unit 
outputting the image based on the condition for outputting the 
image . 

20 The image output apparatus may include an extractor 

extracting the depth information based on a parallactic image 
obtained by photographing the subject from different viewpoints. 

The extractor may extract data of an aimed object from the 
25 image based on the depth information, and the 

condition -determining unit may determine the condition for 
outputting the image based on information including the data of 
the aimed object. 

30 The extractor may extract image information included in the 

image and extracts the aimed object based on the image 
information . 

According to the third aspect of the present invention, an 
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camera comprises: a first input unit inputting a parallactic 
image of a subject photographed from different viewpoints; a 
second input unit inputting a refined image of the subject; and 
a condition-determining unit determining a condition for 
5 inputting the refined image based on the parallactic image. 

The camera may include an extractor extracting a depth 
information indicating the distance to each part of the subject 
based on the parallactic image, wherein the 
10 condition-determining unit may determine the condition for 
inputting the refined image based on the depth information. 

The condition for inputting the refined image may include 
at least one of the conditions of focal length, aperture 
15 condition, exposure time, gradation characteristics, and 
sensitivity. 

The extractor may extract data of an aimed object from the 
image based on the depth information, and the 
20 condition-determining unit may determine the condition for 

inputting the refined image based on information including the 
data of the aimed object. 

The extractor may extract image information included in the 
25 refined image and extracts the aimed object based on the image 
information. 

According to the fourth aspect of the present invention, 
a method of processing an image obtained by photographing a 
30 subject, comprises determining a condition for processing the 
image based on depth information indicating the distance to each 
part of the subject. 

According to the fifth aspect of the present invention, a 
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method of output ting an image obtained by photographing a 
subject, comprises determining a condition for outputting the 
image based on depth information indicating the distance to each 
part of the subject. 

5 

According to the sixth aspect of the present invention, a 
method of inputting a refined image of a subject, comprises: 
inputting a parallactic image of the subject photographed from 
different viewpoints; determining a condition for inputting the 
10 refined image based on the parallactic image; and inputting the 
refined image of the subject based on the condition for inputting 
the refined image. 

The method may include inputting a raw image of the subject 
15 prior to the determination of the condition for inputting the 
refined image, wherein the determination of the condition for 
inputting the refined image may determine the condition based on 
the raw image of the subject. 

20 According to the seventh aspect of the present invention, 

a recording medium storing therein a program executed by a 
computer to perform a method of processing an image obtained by 
photographing a subject, comprises determining a condition for 
processing the image based on depth information indicating the 

25 distance to each part of the subject. 

According to the eighth aspect of the present invention, 
a recording medium storing therein a program executed by a 
computer to perform a method of outputting an image obtained by 
30 photographing a subject, comprises determining a condition for 
outputting the image based on depth information indicating the 
distance to each part of the subject. 

This summary of the invention does not necessarily describe 
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all necessary features so that the invention may also be a 
sub -combination of these described features. 

BRIEF DESCRIPTION OF THE DRAWINGS 

5 

Fig. 1 shows a camera 10 of the first embodiment according 
to the present invention, 

Fig. 2 is a block diagram of the control unit 50 according 
to the present embodiment, 
10 Fig. 3 is a functional block diagram of the extractor 60, 

Fig. 4 is a functional block diagram of the 
condition-determining unit 70, 

Fig. 5 is a flowchart showing the method of photographing 
a refined image, 

15 Fig. 6 is a flowchart showing in detail the method of 

extracting an aimed object, step 106 in Fig. 5, 

Fig. 7 is a flowchart showing in detail the method of 
determining each of the conditions, step 10 8 in Fig. 5, 

Fig. 8 is a flowchart showing in detail the method of 
20 photographing a refined image, step 110 in Fig. 5, 

Fig. 9 shows a camera of the second embodiment according 
to the present invention. 

Fig. 10 is a block diagram of the control unit 5 0 according 
to the present embodiment , and 
25 Fig. 11 shows a Laboratory system of the third embodiment 

according to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

30 The invention will now be described based on the preferred 

embodiments, which do not intend to limit the scope of the present 
invention, but exemplify the invention. All of the features and 
the combinations thereof described in the embodiment are not 
necessarily essential to the invention. 
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Fig. 1 shows an electronic camera 10 of the first embodiment 
according to the present invention. The camera 10 photographs 
a raw image of a subject and then photographs a refined image of 
5 the subject based on information from the raw image. The refined 
image is processed based on a process condition determined based 
on information from the raw image. The refined image is then 
output based on a condition determined based on information from 
the raw image . 

10 

The camera 10 includes an input unit 20, an A/D converter 
30, a memory 40, a control unit 50, a release button 52, a 
recording unit 90 and an output unit 92. The camera 10 may be, 
for example, a digital still camera or a digital video camera. 

15 

The input unit 2 0 includes a parallactic image data input 
unit 22 and a normal image data input unit 24. The parallactic 
image data input unit 22 inputs a parallactic image, which is 
photographed from different viewpoints. The parallactic image 

20 data input unit 22 has a parallactic lens 32, a parallactic 
shutter 34, and a parallactic charge coupled device (CCD) 36. The 
parallactic lens 32 forms an image of a subject. The parallactic 
shutter 34 has a plurality of shutter units each of which serves 
as viewpoints. The parallactic shutter 34 opens one of the 

25 shutter units. The parallactic CCD 36 receives the image of the 
subject through the parallactic lens 32 and whichever of the 
shutter units of the parallactic shutter 34 that is opened. The 
parallactic CCD 36 also receives another image of the subject 
through the parallactic lens 32 and another of the shutter units 

30 of the parallactic shutter 34, which is opened at this time. The 
images received through the parallactic lens 32 and the 
parallactic shutter 34 form a parallactic image. Thus, the 
parallactic CCD 36 receives the parallactic image of the subject 
formed by the parallactic lens 32 and converts it to electronic 
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signals . 

The normal image data input unit 24 inputs a normal image 
photographed from a single viewpoint. The normal image includes 
5 a raw image which was previously photographed, and a refined image 
which is photographed based on an input condition determined in 
accordance with information from the raw image. The normal image 
data input unit 24 has a lens 25, a lens stop 26, a shutter 27, 
a color filter 28 and a charge coupled device (CCD) 29. The lens 
10 25 forms an image of a subject. The lens stop 26 adjusts an 
aperture condition. The shutter 27 adjusts exposure time. The 
color filter 2 8 separates RGB components of the light received 
through the lens 25. The CCD 29 receives the image of the subject 
formed by the lens 25 and converts it to electric signals. 

15 

The A/D converter 3 0 receives analog signals from the 
parallactic image data input unit 22 and the- normal image data 
input unit 24. The A/D converter 30 converts the received analog 
signals to digital signals and outputs the digital signals to the 
20 memory 40. The memory 40 stores the input digital signals. This 
means that the memory 40 stores the data for the parallactic image 
of the subject photographed by the parallactic image data input 
unit 22, and the data for the normal image of the subject 
photographed by the normal image data input unit 24. 

25 

The control unit 50 determines at least one of the 
conditions including an input condition for inputting a refined 
image, a process condition for processing the input image, and 
an output condition for outputting the processed image, based on 
30 information from the parallactic image and the raw image. The 
control unit 50 controls processing of the input image of the 
subject and outputs the processed image. The control unit 50 
controls at least one of the following conditions: focus 
condition of the lens 25, aperture condition of the lens stop 26, 
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exposure time of the shutter 27, output signal of the CCD 29, 
condition of the parallactic shutter 34, and output signal of the 
parallactic CCD 36. 

5 The release button 52 outputs a signal to the control unit 

50 for starting the photographing operation . This means that 
when a user of the camera 10 pushes the release button 52, the 
signal is output to the control unit 50. The control unit 50 then 
controls the input unit 20 to photograph the subject. 

10 

The recording unit 90 records the image and information 
related to the image output from the control unit 50 on a recording 
medium. The recording medium may be, for example, a magnetic 
recording medium such as a floppy disk, or a nonvolatile memory 
15 such as flash memory. 

The output unit 92 outputs the image recorded on the 
recording medium. The output unit 92 may be, for example, a 
printer or a monitor. The output unit 92 may be a small liquid 
20 crystal display (LCD) of the camera 10. In this case, the user 
can see the image processed by the control unit 50 immediately 
after photographing the image . The output unit 92 may be an 
external monitor connected to the camera 10. 

25 Fig. 2 is a block diagram of the control unit 50 according 

to the first embodiment. The control unit 50 includes an image 
pickup control unit 56, an image forming control unit 58, an 
extractor 60, a condition-determining unit 70 and an 
image-processing unit 84. 

30 

The extractor 60 receives a parallactic image photographed 
by the parallactic image data input unit 22 and a normal image 
photographed by the image data input unit 24, from the memory 40. 
As described above, the normal image includes a raw image and a 
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refined image. The extractor 60 extracts an aimed object from 
the normal image based on the information obtained from the 
parallactic image and the normal image. The information includes 
image information of the normal image and depth information of 
5 the parallactic image. The aimed object defined here is an 
independent object at which a photographer aims when 
photographing. The aimed object may be, for example, a person 
in a room when the person and the objects in the room are 
photographed, a fish in an aquarium when the fish and the aquarium 
10 are photographed, or a bird stopping on a branch of a tree when 
the bird and the tree are photographed. The aimed object may be 
an area extracted for a certain purpose. The extractor 60 then 
outputs the information related to the aimed object to the 
condition-determining unit 70. 

15 

The condition-determining unit 7 0 determines at least one 
of the following conditions including an input condition for 
inputting a refined image, a process condition for processing the 
input image, and an output condition for outputting the 
20 input /processed image, based on the information related to the 
image information for the aimed object or the depth information. 

The image forming control unit 58 controls the input unit 
2 0 to form a refined image of the subject based on the input 

25 condition determined by the condition- determining unit 70. This 
means that the image forming control unit 58 controls at least 
one of the conditions including focus condition of the lens 25, 
aperture condition of the lens stop 26, exposure time of the 
shutter 27, and condition of the parallactic shutter 34, based 

30 on the input condition. 

The image pickup control unit 56 controls the input unit 
2 0 to photograph a refined image of the subject based on the input 
condition determined by the condition-determining unit 70. This 
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means that the image pickup control unit 56 controls at least one 
of the conditions including output signal of the CCD 2 9 and output 
signal of the parallactic CCD 36, based on the input condition. 
The output signal of the CCD 2 9 determines the gradation 
5 characteristics based on a gamma (y) curve and sensitivity. The 
image pickup control unit 56 also controls the image -processing 
unit 84 to process the input normal image. 

The image-processing unit 84 receives the normal image 
10 photographed by the image data input unit 24 from the memory 40. 
The image -processing unit 84 then processes the normal image 
based on the processing condition determined by the 
condition-determining unit 70. 

15 Fig. 3 is a functional block diagram of the extractor 60. 

The extractor 60 includes a depth information extractor 62, an 
image information extractor 64 and an aimed object extractor 66. 

The depth information extractor 62 extracts the depth 
20 information indicating the distance to each of components of the 
subject, based on the data for the parallactic image received from 
the memory 40. This means that the depth information extractor 
62 determines a corresponding point for each of the components 
based on the parallactic image and gives a parallax amount. The 
25 depth information extractor 62 extracts the depth information 
based on the parallax amount of each of the components. 
Determining the corresponding point is a known technique, thus 
the explanation of this technique will be omitted. Extracting 
the depth information based on the parallax amount is also a known 
30 technique using the principle of triangulation, thus the 
explanation of this technique will also be omitted. 

The image information extractor extracts the image 
information for the normal image, from the data for the normal 
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image received from the memory 40 . The image information includes, 
for example, data from the normal image such as luminescence 
distribution, intensity distribution, color distribution, 
texture distribution, and motion distribution. 

5 

The aimed object extractor 66 extracts data for the aimed 
object based on the depth information and the image information. 
Each of the images may include, for example, a plurality of 
components. The aimed object extractor 66 recognizes each of the 
10 components based on the depth information. The aimed object 
extractor 66 then specifies the aimed object by referring to the 
depth information and the image information for each of the 
components. The method of specifying the aimed object will be 
described in the following. 

15 

Since cameras are usually used to photograph human beings, 
the aimed object should be related to the photographed person, 
especially to face of the person. Thus, the aimed object in this 
embodiment is targeted to be the face of the photographed person. 

20 Therefore, at first, a component including the face is specified 
depending on assumptions such as "the person should be close to 
the camera" , ""the person should be in the middle of the image" , 
or "the proportional relationship of the height of the person to 
the width and height of the image should be within a predetermined 

25 range" . The distance from the camera to each of the components 
in the image is evaluated based on the depth information. The 
distance from the center of the image to each of the components 
in the image, and the proportional relation ship of the height 
of the components are evaluated based on the image information. 

30 Each of the values is multiplied by predetermined constants 
corresponding to each condition. The multiplied values are added 
for each of the components. The added values are defined as 
weighted averages. The component having a largest weighted 
average is extracted as the component including the aimed object. 
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The constants by which the values for each of the components 
are multiplied may be predetermined based on the aimed object. 
In this embodiment, for example, the aimed object is assumed to 
5 be the face of the photographed person. Therefore, the aimed 
object extractor 66 specifies the area having a skin color as the 
face, based on the image information. The colors of each of the 
components are evaluated based on the color distribution of the 
images. The values of the color distribution may also be 
10 multiplied by predetermined constants and the multiplied values 
are added for each of the components to give the weighted averages. 
The aimed object extractor 66 then outputs the data of the aimed 
object to the condition-determining unit 70. 

15 As described above, the aimed object extractor 66 extracts 

an aimed object based on the depth information in addition to the 
image information. Therefore, even when a plurality of people 
are photographed in the image and their faces are close to each 
other, the faces of the different people can be distinctively 

20 extracted. The aimed object extractor 66, extracting the aimed 
object based on the depth information obtained from the 
parallactic image, can extract the aimed object more precisely 
compared with an extractor using a multi -point distance measuring 
method. 

25 

Fig. 4 is a functional block diagram of the 
condition-determining unit 70. The condition-determining unit 
70 includes an input-condition-determining unit 76, a 
processing-condition-determining unit 74, and an 
30 output-condition-determining unit 72. 

The input -condition-determining unit 76 determines an 
input condition for inputting a refined image, based on the depth 
information and the image information for the aimed object. The 
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input condition for inputting a refined image may be determined 
by considering information related to the camera itself. The 
information related to the camera includes focus condition of the 
lens 25, aperture condition of the lens stop 2 6 and light exposure 
5 measured by the exposure meter, not shown in the drawings, when 
the camera 10 is photographing a raw image. Examples of the 
conditions for inputting a refined image will be explained in the 
following. 

10 The input condition for inputting a refined image may 

relate to the aperture condition of the lens stop 2 6 and exposure 
time of the shutter 27. The input -condition-determining unit 76 
evaluates the distance from the camera 10 to each component of 
the subject, based on the depth information. The 

15 input-condition-determining unit 76 then obtains the maximum 
distance between the components, which is the distance between 
the component nearest to the camera 10 and the component farthest 
from the camera 10. The input-condition-determining unit 76 
determines the aperture condition of the lens stop 2 6 in 

20 accordance with the maximum distance between the components. The 
input -condition-determining unit 76 then determines exposure 
time of the shutter 27 based on the aperture condition of the lens 
stop 26. In the case of photographing a person relatively close 
to the camera with a background including a mountain far from the 

25 camera, for example, the aperture may reduce and the exposure time 
may be long. 

The input condition for inputting a refined image may 
relate to the contrast or white balance of the image. The input 
30 condition in this case is determined based on the data for the 
aimed object. When photographing with a flashlight, the aimed 
object that is supposed to be close to the camera 10 may have a 
good color condition due to the light. However, the components 
other than the aimed object may be dark because these components 
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cannot receive enough light. Therefore, the 

input-condition-determining unit 76 separately determines the 
contrast condition for the aimed object and for the components 
other than the aimed object. 

5 

The input condition for inputting a refined image may 
relate to the focus condition of the lens 25. The input condition 
in this case is determined based on the data for the aimed object . 
The input -condition-determining unit 76 determines the focus 
10 condition such that only the aimed object is focused. 

The image forming control unit 58 controls focal length, 
aperture condition, and the exposure time of the input unit 20, 
based on the input condition determined by the 

15 input -condition-determining unit 76. The image pickup control 
unit 56 controls gradation characteristics and sensitivity of the 
input unit 20, based on the input condition determined by the 
input-condition-determining unit 76. The image data input unit 
24 inputs a refined image based on the input condition determined 

20 by the input -condition -determining unit 76. 

The recording unit 90 may record the refined image 
including data for the input condition, when recording the data 
for the refined image on the recording medium. The output unit 
25 92 may output the refined image including the data for the input 
condition, when outputting the data for the refined image to an 
external monitor or an external printer. 

The processing-condition-determining unit 74 determines a 
30 process condition for processing a normal image, based on the 
depth information and the image information for the aimed object . 
The normal image includes a refined image and a raw image. 
Examples of possible process conditions for processing a normal 
image are explained in the following. 
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The process condition for processing a normal image may 
relate to color of the image. The process condition in this case 
is determined based on the depth information. The 
5 processing-condition-determining unit 74 determines the 

condition by which the components close to the camera 10 are 
highly colored and the components far from the camera 10 are less 
colored. With this condition, the position of each of the 
components can be naturally expressed. The process condition in 
10 this case may be determined based on the data for the aimed obj ect . 
The processing- condition-determining unit 74 may, for example, 
separately determine the color condition for the aimed object and 
for the components other than the aimed object so that all the 
components have optimum gradation. 

15 

The process condition for processing a normal image may 
relate to compression of the image. The process condition in this 
case is determined based on the data for the aimed object. The 
processing-condition-determining unit 74 separately determines 

20 the compression condition of the image for the aimed object and 
for the components other than the aimed object, so that the aimed 
object is not deteriorated, even though the data size of the image 
itself is compressed. The processing-condition-determining unit 
74 may separately determine the color compression condition for 

25 the aimed object and components other than the aimed object. 



The processing-condition-determining unit 74 may 
determine a process condition by which the aimed object in the 
image is magnified and the magnified aimed object is composited 
30 with a background image. The background image may be the 

components included in the original image other than the aimed 
object, or an image previously selected by the user of the camera 
10 . 
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The image processing unit 84 processes the normal image 
based on the process condition determined by the 
processing-condition-determining unit 74 . The recording unit 90 
may record the processed image along with data for the process 
5 condition when recording the data for the processed image on the 
recording medium. The output unit 92 may output the processed 
image along with the data for the process condition when 
outputting the data for the processed image to an external monitor 
or an external printer. 

10 

The output-condition-determining unit 72 determines an 
output condition for outputting a normal image, based on the depth 
information and the image information for the aimed object. The 
normal image includes a refined image, a raw image and a processed 
15 image thereof. Examples of the output condition for outputting 
a normal image will be explained in the following. 

The output condition for outputting a normal image may 
relate to a condition for displaying the image . It takes a certain 
20 amount of time to display an image having a relatively large data 
size on a monitor. Therefore, the output-condition-determining 
unit 72 determines the output condition by which the aimed object 
is displayed antecedent to the rest of the components other than 
the aimed object. 

25 

The output condition for outputting a normal image may 
relate to a condition for transferring the image when the image 
is a motion picture. When the camera 10 photographs a motion 
picture and transfers the motion picture through a communication 
30 network to be displayed on a monitor connected to the network, 
the output -condition-determining unit 72 may determine the 
output condition for transferring data for the motion picture. 
For example, the data for the aimed object is transferred at a 
high transfer rate and the data for the rest of the components 
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are transferred at a low transfer rate in accordance with the 
output condition. Furthermore, the 

output-condition-determining unit 72 may determine an output 
condition for the frequency modulation in which the acuteness of 
the aimed object is emphasized and the acuteness of the rest of 
the components is weakened. 

The output condition for output ting a normal image may 
relate to a condition for printing the image. The 
output-condition-determining unit 72 may determine the output 
condition by which the image is developed, based on the condition 
of the aimed object. The output-condition-determining unit 72 
may separately determine the output condition for the aimed 
object and for the components other than the aimed object, so that 
the aimed object does not deteriorate, even though the gradation 
of the image itself is varied in accordance with the dynamic range 
when the image is output to a printer or a monitor. 

The recording unit 90 may record the normal image along with 
at least one of data for the input condition, data for the process 
condition, data for the output condition, and the depth 
information, on the recording medium. 

As described above, the camera 10 of this embodiment is 
capable of determining the input condition for inputting a 
refined image, the process condition for processing an image, and 
the output condition for outputting an image, based on the depth 
information or the image information for the aimed object. 
Therefore, each of the conditions can be precisely determined. 

Fig. 5 is a flowchart showing the method of outputting a 
refined image. When the user of the camera 10 pushes the release 
button 52, the camera 10 starts photographing (SlOO) . At this 
time, the parallactic image data input unit 22 inputs a 
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parallactic image (S102) . At the same time, the image data input 
unit 24 inputs a raw image (S104) . Then, the aimed object 
extractor 66 extracts an aimed object from the raw image (S106) . 
The condition-determining unit 70 determines an input condition 
5 for inputting a refined image, a process condition for processing 
the input image, and an output condition for outputting the input 
image (S108) . The input unit 20 then inputs a refined image in 
accordance with the input condition determined by the 
condition-determining unit 70 (SllO) . 

10 

The image processing unit 84 processes the input refined 
image based on the process condition determined by the 
condition -determining unit 70 (S112) . The recording unit 90 
records the refined image along with data for at least one of the 
15 input condition, the process condition and the output condition 
(S114) . The output unit 92 outputs the image recorded by the 
recording unit 90 in accordance with the output condition 
determined by the condition-determining unit 70 (S116) . The 
operation for outputting the refined image is terminated (S118) . 

20 

Fig. 6 is a flowchart showing in detail the method of 
extracting an aimed object, step 106 in Fig. 5. The depth 
information extractor 62 extracts depth information based on the 
parallactic image (S120) . The image information extractor 64 
25 extracts image information based on the raw image (S122) . The 
aimed object extractor 66 extracts an aimed object based on the 
depth information and the image information (S124) . 

Fig. 7 is a flowchart showing in detail the method of 
30 determining each of the conditions, step 108 in Fig. 5. The 
input-condition-determining unit 76 determines an input 
condition for inputting a refined image (S140) . The 
processing-condition-determining unit 74 determines a process 
condition for processing an image (3142) . The 
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output-condition-determining unit 72 determines an output 
condition for outputting an image (S144) . 

Fig. 8 is a flowchart showing in detail the method of 
5 photographing a refined image, step 110 in Fig. 5. The image 
forming control unit 58 controls at least one of the conditions 
including focal length, aperture condition, and exposure time of 
the input unit 2 0 based on the input condition determined by the 
condition-determining unit 70 (S160) . The input unit 20 then 
10 forms an image of a subject. The image pickup control unit 56 
controls at least either of the conditions including gradation 
characteristic and sensitivity of the input unit 20 (S162) . The 
input unit 2 0 photographs the image of the subject as a refined 
image . 

15 

Fig. 9 shows a camera 110 of the second embodiment according 
to the present invention. The camera 110 previously photographs 
a raw image of a subject and then determines an input condition 
for photographing a refined image, and an output condition for 

20 outputting the input image, based on the raw image. The camera 
110 in this embodiment is a silver halide type camera by which 
an image of a subject is formed on a silver halide film. The camera 
110 includes an input unit 120, an A/D converter 30, a memory 40, 
a control unit 150, and a release button 52. The A/D converter 

25 30, the memory 40, and the release button 52 in this embodiment 
have same structures and functions as those explained in the first 
embodiment. Therefore, the explanation of these parts will be 
omitted . 

30 The input unit 120 includes a parallactic image data input 

unit 122, a raw image data input unit 124 for inputting an 
electronic raw image and a refined image data input unit 130 for 
forming a refined image on a film.. The parallactic image data 
input unit 122 and the raw image data input unit 124 in this 
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embodiment respectively have the same structures and functions 
as the parallactic image data input unit 22 and the image data 
input unit 24 explained in the first embodiment. Therefore, the 
explanation of these parts will be omitted. 

5 

The refined image data input unit 13 0 includes a lens 132, 
a lens stop 134, a shutter 136 and a photographing unit 138. The 
lens 132, the lens stop 134 and the shutter 136 in this embodiment 
respectively have the same structures and functions as the lens 
10 25, the lens stop 26 and the shutter 27 shown in Fig. 1 of the 
first embodiment. The photographing unit 13 8 receives an optical 
image of a subject and forms an image of the subject on silver 
halide film. 

15 The image data input unit 24 of the first embodiment inputs 

both a raw image and a refined image. However, as for the camera 
110 of this embodiment, the raw image data input unit 124 inputs 
an electronic raw image, and the refined image data input unit 
13 0 inputs a refined image and forms the refined image on a film. 

20 

Fig. 10 is a block diagram of the control unit 15 0 according 
to the second embodiment. The control unit 150 in this embodiment 
includes an extractor 60, a condition-determining unit 70, an 
image forming control unit 58, and an image pickup control unit 
25 56. The extractor 60 in this embodiment has the same structure 
and function as the extractor 6 0 of the first embodiment, thus 
the explanation of this part will be omitted. 

The condition-determining unit 70 determines at least one 
30 of the conditions including the input condition and the output 
condition, based on the depth information and the image 
information of the aimed object extracted by the extractor 60. 

The image forming control unit 58 controls focal length. 
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aperture condition, and the exposure time of the input unit 12 0 
based on the input condition for inputting a refined image 
determined by the input -condition-determining unit 76 . The image 
pickup control unit 56 controls gradation characteristics and 
5 sensitivity of the input unit 12 0, based on the input condition 
for inputting a refined image determined by the 
input-condition-determining unit 76. The refined image data 
input unit 13 0 forms an image of a subject on silver halide film 
along with information of the output condition or the depth 
10 information. A space is provided in the film, for the information 
related to the images . 

In this embodiment, the camera 110 includes the raw image 
data input unit 124 for inputting an electronic raw image, in 
15 addition to the image data input unit 130 for inputting a refined 
image. Therefore, the camera can automatically set an optimum 
condition for photographing a refined image -of the subject. 

Fig. 11 shows a laboratory system of the third embodiment 
20 according to the present invention. The laboratory system 

includes a reading unit 210, a memory 22 0, a control unit 23 0, 
a recording unit 240 and an output unit 250. 

The reading unit 210 reads data of images and information 
25 related to the images from a recording medium for a digital camera, 
or a film. The recording medium for a digital camera may be a 
floppy disk or a removable flash memory. The reading unit 210 
may be a floppy disk drive, for example, and reads the data of 
the images and the information from the recording medium for a 
30 digital camera. The reading unit 210 may have a communication 
unit through which data of an image photographed by a digital 
camera and information related to the image are transmitted over 
a computer network such as the Internet. Thus, the reading unit 
210 can read the data of the images and the information thereof. 
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The reading unit 210 may be a film scanner, for example, 
and may read the data of the images and the information thereof 
from a film. A space is provided in the film, for the information 
5 related to the images. The reading unit 210 reads data of the 
images and the information related to each of the images. 

The memory 22 0 stores the data of the images and the 
information thereof read by the reading unit 210. In this 
10 embodiment, the memory 220 stores depth information and image 
information as the information related to the images. 

The control unit 23 0 has the same function as the control 
unit 50 of the camera 10 according to the first embodiment. The 
15 control unit 50 determines at least either of the conditions for 
processing or outputting the image, based on the depth 
information and the image information stored in the memory 220. 

As for the films, the space for information related to the 
20 images is not large, therefore, only the image information 
related to the aimed object may be written. In this case, the 
condition for processing or outputting the image is determined 
based on the image information of the aimed object written on the 
film. As for the recording medium of the digital camera, 
25 parallactic images may be stored in the recording medium, and the 
depth information may be extracted based on the parallactic 
images. 

The control unit 23 0 processes an image stored in the memory 
30 220, based on the process condition. The processed image is 
recorded on the recording unit 240. The image recorded on the 
recording unit 240 is output from the output unit 250. The output 
unit 250 may be a color laser printer, or a digital printer. 
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The laboratory system in this embodiment may be a 
workstation, or a system using a computer such as a personal 
computer. The computer serves as the memory 220, the control unit 
23 0 and the recording unit 24 0 shown in Fig. 11. Peripheral 
5 apparatus that together compose the system with the computer, may 
serve as the reading unit 210 and the output unit 2 50 shown in 
Fig. 11. 

As for the control unit 230, the method of determining the 
10 process condition based on the depth information may be provided 
with a software program stored in a recording medium 300 shown 
in Fig. 11. Similarly, the method of determining the output 
condition based on the depth information may be provided with a 
software program stored in a recording medium 3 00 shown in Fig. 
15 11. 

As described above, the aimed object extractor 66 extracts 
an aimed object based on the depth information and the image 
information. Therefore, even when a plurality of people are 
20 photographed in the image and their faces are close to each other, 
the faces of the different people can be distinctively extracted. 

Furthermore, the input condition, the process condition 
and the output condition are determined based on the depth 
25 information or the image information of the aimed object. 

Therefore, each of the conditions are precisely determined for 
the aimed object and the components in the image other than the 
aimed object. 

30 Furthermore, as the aimed object extractor 66 extracts the 

aimed object based on the depth information obtained from the 
parallactic image, the aimed object can be extracted more 
precisely compared with the method using multi-point distance 
measurement . 
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Although the present invention has been described by way 
of exemplary embodiments, it should be understood that many 
changes and substitutions may be made by those skilled in the art 
5 without departing from the spirit and the scope of the present 
invention which is defined only by the appended claims. 
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WHAT IS CLAIMED IS: 

1. An image processing apparatus for processing an image 
obtained by photographing a subject, comprising a 
condition-determining unit determining a condition for 
processing said image based on depth information indicating the 
distance to each part of said subject. 

2. An image processing apparatus as set forth in claim 1, 
further comprising an image processing unit processing said image 
based on said condition for processing said image. 

3. An image processing apparatus as set forth in claim 1, 
further comprising an extractor extracting said depth 
information based on a parallacbic image obtained by 
photographing said subject from different viewpoints. 

4. An image processing apparatus as set forth in claim 3, 
wherein said extractor extracts data of an aimed object 

from said image based on said depth information, and 

said condition- determining unit determines said condition 
for processing said image based on information including said 
data of said aimed object. 

5. An image processing apparatus as set forth in claim 4, 
wherein said extractor extracts image information included in 
said image and extracts said aimed object based on said image 
information . 

6. An image output apparatus for outputting an image obtained 
by photographing a subject, comprising a condition-determining 
unit determining a condition for outputting said image based on 
depth information indicating the distance to each part of said 
subj ect . 
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7. An image output apparatus as set forth in claim 6, further 
comprising an output unit outputting said image based on said 
condition for outputting said image. 

8. An image output apparatus as set forth in claim 6, further 
comprising an extractor extracting said depth information based 
on a parallactic image obtained by photographing said subject 
from different viewpoints. 

9. An image output apparatus as set forth in claim 8, 
wherein said extractor extracts data of an aimed object 

from said image based on said depth information, and 

said condition-determining unit determines said condition 
for outputting said image based on information including said 
data of said aimed object. 

10. An image output apparatus as set forth in claim 9, wherein 
said extractor extracts image information included in said image 
and extracts said aimed object based on said image information. 

11. A camera comprising: 

a first input unit inputting a parallactic image of a 
subject photographed from different viewpoints ; 

a second input unit inputting a refined image of said 
subj ect ; and 

a condition-determining unit determining a condition for 
inputting said refined image based on said parallactic image. 

12. A camera as set forth in claim 11, further comprising an 
extractor extracting a depth information indicating the distance 
to each part of said subject based on said parallactic image, 

wherein said condition-determining unit determines said 
condition for inputting said refined image based on said depth 
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information . 

13. A camera as set forth in claim 11, wherein said condition 
for inputting said refined image includes at least one of the 
conditions of focal length, aperture condition, exposure time, 
gradation characteristics, and sensitivity. 

14. A camera as set forth in claim 12, wherein said extractor 
extracts data of an aimed object from said image based on said 
depth information, and 

said condition- determining unit determines said condition 
for inputting said refined image based on information including 
said data of said aimed object. 

15. A camera as set forth in claim 14, wherein said extractor 
extracts image information included in said refined image and 
extracts said aimed object based on said image information. 

16. A method of processing an image obtained by photographing 
a subject, comprising: determining a condition for processing 
said image based on depth information indicating the distance to 
each part of said subject. 

17. A method of outputting an image obtained by photographing 
a subject, comprising: determining a condition for outputting 
said image based on depth information indicating the distance to 
each part of said subject. 

18. A method of inputting a refined image of a subject, 
comprising : 

inputting a parallactic image of said subject photographed 
from different viewpoints; 

determining a condition for inputting said refined image 
based on said parallactic image; 
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inputting said refined image of said subject based on said 
condition for inputting said refined image. 

19. A method as set forth in claim 18 further comprising 
inputting a raw image of said subject prior to said determination 
of said condition for inputting said refined image, 

wherein said determination of said condition for inputting 
said refined image determines said condition based on said raw 
image of said subject. 

20. A recording medium storing therein a program executed by 
a computer to perform a method of processing an image obtained 
by photographing a subject, comprising: determining a condition 
for processing said image based on depth information indicating 
the distance to each part of said subject. 

21. A recording medium storing therein a program executed by 
a computer to perform a method of output ting an image obtained 
by photographing a subject, comprising: determining a condition 
for outputting said image based on depth information indicating 
the distance to each part of said subject. 
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ABSTRACT OF THE DISCLOSURE 

A camera includes a release button, an input unit, an A/D 
converter, a memory, a control unit, a recording unit and an 
5 output unit. The release button outputs a signal to instruct the 
input unit to photograph an image of a subject. The memory stores 
data of the image converted by the A/D converter. The control 
unit determines an input condition, a process condition, and an 
output condition for the image to give a refined image. The 
10 recording unit records the refined image on a recording medium. 
The output unit outputs the refined image. 
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